Repository navigation
Hold MLOps artifacts on Unity Catalog volumes through the Files API - #114
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
Walkthrough
Merge Risk:
|
Change map
|
cab9cb7 to
91d494f
Compare
044ad8c to
b322307
Compare
b322307 to
d1d6d05
Compare
d1d6d05 to
3f3d201
Compare
3f3d201 to
98fb53e
Compare
98fb53e to
e2d71ad
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…FilesystemSink Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…equality Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tream log Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ep only table writes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CSV exports open the stream without a size, so anything over 256 MiB rolled over to .chunks parts that Spark cannot read as one file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The export preflight lists the destination before writing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pass an explicit 16 MiB part size and 4 upload threads to files.upload when the SDK accepts them (databricks-sdk 0.70+), so a large plain object no longer buffers up to parallelism x part_size with the SDK's defaults. Older SDKs upload single-threaded in 10 MiB chunks and get no extra args. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fb03edd to
01ca25a
Compare
**Stack:** main (#53 merged) → #112 → #114 → **#50** → #60 → #64 → #115. Merge bottom to top. ## Summary Every graph snapshot, training table and model bundle now carries a self-contained `_provenance.json` that leads back to the source table versions (and locations, wherever the source backend can name them), query, plan and engine build that produced it. Today a model has no id, an artifact records no lineage, an index works only on the machine that built it, and scoring needs the training job's `Graph`. This PR adds `kumo.MLOps` / `with kumo.mlops(...)` to store each artifact once under one artifact root (`s3://`, `gs://`, local, or a `(connector, path)` file store from #112); `GraphSnapshot.export` / `load_from_connector` to move a checksummed index between machines; `snapshot.source_changes(graph)` to see which tables changed since a snapshot; and `Trainer.predict(snapshot)` to score a stored snapshot with the trained query. `MLOps(on_error='raise')` turns a failed store into a `RuntimeError` after the call finishes. Off unless asked for. **Design decision:** for this release MLOps is opt-in per call or per block (`mlops=` or `with kumo.mlops(...)`). A call outside both runs without MLOps, and `on_error="raise"` does not detect that. A model- or workflow-bound association with an explicit `mlops=False` was considered and is left as a follow-up. See "Per call, not per model" in `docs/mlops.md`. ## Usage ```python cfg = kumo.MLOps("s3://<bucket>/kumo") # store=True: snapshots, tables, models with kumo.mlops(cfg): # or mlops=cfg on one call snapshot = graph.snapshot() result = trainer.fit(pq.generate_training_table(snapshot), snapshot=snapshot) # Later, on another machine with no Graph code: snapshot = GraphSnapshot.load_from_connector(cfg, id=result.snapshot_id) trainer = Trainer.load_from_connector(cfg, path=f"models/{result.id}") scores = trainer.predict(snapshot) ``` ## Back-compat Without a block or `mlops=`, existing calls behave as before. | Situation | Behaviour | |---|---| | Index built before this PR | Reused, not rebuilt; its manifest is backfilled with `"reconstructed": true`. | | Bundle published before this PR | Loads and predicts identically; id computed at load, `manifest` is `None`. | | Artifact from this PR read by an older engine | Extra files are ignored. | | A single S3 or GCS object source | Versioned by VersionId or entity tag, so its snapshot gets a new id once. | | `verify_model_bundle` return value | 64 hex instead of a 16-hex digest no caller used. | | `DatabricksConnector(identity=...)` | Unchanged; `identity` is now optional (ambient auth). | | Manifest, sidecar or root write fails | A warning; the call completes as before. A failed store raises a `RuntimeError` instead, once the call has finished, with `MLOps(on_error='raise')`. | ## Testing - kumo-engine: `test_snapshot_export.py` (round trip, reuse, interrupted and concurrent exports, corrupt markers, `source_changes`, `on_error`), `test_snapshot_manifest.py`, `test_training_table_export.py`, `test_holdout_df.py` (published model's holdout, loaded and fetched holdout, empty test split), `test_bundle_provenance.py`, `test_query_from_snapshot.py`, `test_filestore_sink.py` (identical bundle reused), `test_mlops.py`, `test_graph_mismatch.py`, `test_artifacts.py`. - kumo-core typed source versions, bundle ids and export sidecars; kumo-s3 `test_sink_delete.py`; Snowflake, Databricks and BigQuery identity tests assert `version_kind`. - DiskGraph: Rust `range_checksums_cover_every_listed_file_and_the_manifest`; pybind `test_exception_exports.py`. - Env-gated live S3 and GCS round trips in `test_snapshot_export_live.py`. Change map: see the `kumo-pr-change-map` comment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Stack: main (#53 merged) → #112 → #114 → #50 → #60 → #64 → #115. Sits between #112 and #50. Merge bottom to top.
Summary
A
/Volumes/<catalog>/<schema>/<volume>/...path is reachable only through the FUSE mount on Databricks compute, so a training job on a laptop, EC2 box or CI cannot publish a model or export scores to the volume a Databricks job reads. This PR makes such a path a file destination ofDatabricksConnectorfrom any machine. The sink implements #112'sFileStoreSinkand reads and writes the volume through the Databricks Files API with the connector's identity, reusing #112's writers. Files up to 4 GiB are plain objects that Spark reads directly; larger ones are stored as 4 GiB parts (the Files API caps one upload at 5 GiB). No SQL warehouse is used.Usage
The identity needs
USE CATALOG,USE SCHEMA,READ VOLUME(load) andWRITE VOLUME(export).Back-compat
DatabricksConnectorlocation=)/Volumes/...path on aDatabricksConnector/Volumes/...string root on Databricks computeFileConnectorover FUSE.Output directory '/Volumes/.../scores/' already contains data from another export. Choose an empty destination path.FileNotFoundError,PermissionErrororOSErrorwith the volume path; no signed URL in the message. A failed write cancels its upload, so nothing is committed.<name>.chunks/<generation>/parts; only kumo reads it as one file.Testing
kumo-databricks/tests/test_volume.py(10 tests, in-memory Files API intests/fakes.py): reads across part boundaries; sized writes plain up to a part; failed and refused uploads; a download that ignoresRangeis refused; directories and missing paths; bundle publish and fetch, Parquet and frame exports through a volume path; an occupied directory refused before scoring.kumo.mlopsfit and predict with the artifact root on a volume writes snapshots, models and predictions there.Change map: see the
kumo-pr-change-mapcomment.🤖 Generated with Claude Code